System for processing video and audio information

ABSTRACT

A software development kit prioritizes certain aspects of an acquired image and introduces the prioritized aspects to a main processor. Alternatively, a coprocessor, or the coprocessor and the development kit, manipulate(s) the acquired image and introduce(s) the manipulated image to the processor. The reprogramming of either one of the development kit and the coprocessor may be initiated by either one of them or by the processor and the programming may be provided by the main processor. A central station and a gate array may also be individually reprogrammable by the main processor which sets up, programs and controls an intelligent imaging platform in accordance with the individual reprogramming. A reprogramming of an audio acquisition stage may also be initiated by that stage and any of the other stages and the processor and may be provided by the processor. The audio information may be related to the acquired image.

This is a divisional application of 10/113,480 (attorney's fileIQINV-58963) filed in the United States Patent and Trademark Office onMar. 29, 2002.

BACKGROUND OF A PREFERRED EMBODIMENT OF THE INVENTION

This invention relates to a system for and method of processing anacquired image. More particularly, the invention relates to a systemfor, and method of, processing an image in a wide variety of ways, notpreviously capable of being accomplished in the prior art, to provideresults which are enhanced compared to what has been able to be achievedin the prior art.

BACKGROUND OF THE INVENTION

Systems are now in use for processing an acquired image. For example,systems are now in use for processing an acquired image to determine theentrance into, and the departure of individuals from, a defined areasuch as an enclosure. Systems are also in use for determining theidentity of individuals and objects in an enclosure. Systems are furtherin use for tracking the movement and variations in the positioning ofindividuals in an enclosure. These are only a few examples of differenttypes of processing and uses of acquired images.

As of now, different processing and uses of acquired images requiredifferent types of systems to be constructed. For example, the samesystem cannot be used to identify an individual in a crowd and to trackthe movement of the identified individual in the crowd and particularlythe movement of the individual in a defined area such as an enclosure orfrom one defined area to another defined area. The same system cannotalso be used to magnify a particular portion of an acquired image andprocess that magnified portion. Since different systems are required toperform different functions, costs to individuals or organizations haveincreased, available space has become limited and complexities inoperation have become magnified.

BRIEF DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

A software development kit prioritizes certain aspects of an acquiredimage and introduces the prioritized aspects to a main processor.Alternatively, a coprocessor, or the coprocessor and the developmentkit, manipulate(s) the acquired image and introduce(s) the manipulatedimage to the processor. The reprogramming of either one of thedevelopment kit and the coprocessor may be initiated by either one ofthem and by the processor and the programming may be provided by themain processor.

A central station and a gate array may also be individuallyreprogrammable by the main processor which sets up, programs andcontrols a camera in accordance with the individual reprogrammings.

A reprogramming of an audio acquisition stage may also be initiated bythat stage and any of the other stages and the processor and may beprovided by the processor. The audio information may be related to theacquired image.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 is a diagram, primarily in block form, of a system (hardware andsoftware) constituting a preferred embodiment of the invention forprocessing acquired images on a global basis where the system can beused to process the images in a wide variety of different ways;

FIG. 2 is an expanded diagram, primarily in block form, of the softwareportion of the system shown in FIG. 1;

FIG. 3 is an expanded diagram, partially in block form, showing howdifferent portions of the hardware in FIG. 1 are interconnected bybuses;

FIG. 4 is an expanded diagram, partially in block form, showing howdifferent portions of the system shown in FIGS. 1-3 are reprogrammable,and how the different reprogrammable stages are connected to one anotherby interfaces;

FIG. 5 is a schematic view showing the relationship between areprogrammable stage and a non-reprogrammable stage, the reprogrammablestage being capable of providing a plurality of generic operations and aplurality of custom defined operations and the non-reprogrammable stagebeing capable of providing only a plurality of generic operations;

FIG. 6 is a schematic diagram showing how different primary blocks inthe system shown in FIGS. 1-5 can be combined in different patentablecombinations depending upon the results desired to be obtained from theprocessing of the acquired image and also including a chart showingdifferent combinations of the blocks shown in FIG. 6; and

FIG. 7 is a chart illustratively showing a number of individualfunctions that may be provided by the system shown in FIGS. 1-6 toaccomplish different desired results in processing an acquired image.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

FIG. 1 is a circuit diagram, primarily in block form, of a systemgenerally indicated at 10 and constituting a preferred embodiment of theinvention. The system 10 is shown as being divided by broken lines 12into a central station generally indicated at 14 and by broken lines 16into an intelligent imaging platform generally indicated at 18. Acommunications arrangement formed by one or more communications channelsand generally indicated at 20 is disposed in the intelligent imagingplatform 18. The intelligent imaging platform 18 is in turn indicated bybroken lines 22 as including a software section generally indicated at24. Substantially everything within the broken lines 16 (except for thecommunications arrangement 20 and the software section 24) constituteshardware generally indicated at 28 a, 28 b and 28 c. The hardwaresection 28 a may be considered to include software which interfaces withthe hardware in the section.

The hardware section in FIG. 1 also includes an image acquirer 30 forreceiving an image and converting the image to signals, preferablyelectrical, in a conventional manner. The hardware section also includesan audio codec or audio acquirer 32 for receiving audio informationwhich may be related to the video information. The audio codec oracquirer 32 may include an audio coder decoder designated as a “codec”.Signals pass through a bus 31 between the image acquirer 30 and a fieldprogrammable gate array 34 which may be constructed in a conventionalmanner. Signals also pass through a bus 33 between the gate array 34 anda coprocessor 36. The gate array 34 and the coprocessor 36 are disposedin the hardware section 28 a.

The audio signals preferably pass through a bus 35 between the audiocodec or acquirer 32 and the field reprogrammable gate array 34.However, the audio signals could pass through a bus between the audiocodec or acquirer 32 and the coprocessor 36. The system is more flexiblewhen the audio signals pass between the audio codec or acquirer 32 andthe field reprogrammable gate array 34 than when the audio signals passbetween the audio codec or acquirer 32 and the coprocessor 36. Theability of the signals from the audio acquirer 32 to pass to either thegate array 34 or the coprocessor 36 may be seen by the extension of thebus 35 to the audio/video interface for the hardware section 28 a.

Signals pass between the hardware section 28 a and the software section24 through a bus 38. Signals also pass between a miscellaneousinput/output stage 40 (considered as hardware) and the software section24 through a bus 42. Signals also pass through the hardware section 28 band the software section 24 through a bus 44. The hardware section 28 bincludes a compact flash card interface 46, a PC card interface 48 and aPCI interface 50. The hardware section 28 b provides information storageand includes a capacity for providing information storage expansion andother (non-storage) expansion.

The software section 24 includes a video manipulator 52, an audiomanipulator 54, an event generator 56, an event responder 58, a platformuser interface 60 and a kernel operating system 62. Each of these stageshas an arrow 64 disposed in an oblique direction at the bottom rightcorner of the stage. The oblique arrow 64 indicates that the stage iscapable of being reprogrammed. The reprogramming of any stage with thearrow 64 can be initiated by any stage whether the other stage has thearrow 64 to indicate its capability of being reprogrammed. For example,the reprogramming of any of the stages 34, 36 and 52-62 (even numbersonly) can be self initiated and can be initiated by any of the otherstages 34, 36 and 52-62 and by any other stages such as the stages 70,72 and 74. Thus, each of the stages 34, 36 and 52-62 (even numbers only)is illustratively able to be reprogrammed. Thus, the stages 34, 36,52-62 and 65 (even numbers only) receiving such communications have anenhanced flexibility in operation in comparison to the stages which donot receive such reprogramming. Each reprogrammable stage including thestages 34, 36 and 52-62 (even numbers only) can also initiatereprogramming of itself. The reprogramming of each reprogrammable stageincluding the stages 34, 36 and 52-62 (even numbers only) can beinitiated by almost any stage in the system, except for the imageacquirer 30, the audio acquirer or codec 32, the miscellaneousinput/output 40 and the storage and expansion stage 28 b.

A software development kit 65 is indicated by a cloud designated as“platform configuration” with an arrow 64 in the upper left corner. Theoutput from the software development kit 65 is introduced to a mainprocessor 66 to control the operation of the main processor. Thesoftware development kit may be considered to be within the mainprocessor 66. The main processor 66 reprograms individual ones of thestages 34, 36 and 52-64 (even numbers only) and the software developmentkit 65 to control the image acquired by the stage 30 from the camera andthe audio acquired by the stage 32 from the camera.

The field programmable gate array 34 provides reprogrammable arrays ofgates to clarify and sharpen the video data acquired from the imageacquisition stage 30 and introduces the clarified image to thecoprocessor 36. The coprocessor 36 manipulates the audio and video datadepending upon the results desired to be obtained from the system 10.For example, different manipulations may be provided by the coprocessor36 when the image is targeted on a single person or a group of people oron an inanimate object. The miscellaneous input/output stage 40 providessuch information as motion sensing to indicate to an alarm panel thatthe camera has observed and detected motion in a scene. The hardwaresection 40 can also indicate to the camera that some external device hasdetected motion and wishes to inform the camera that an event worthobserving is taking place. In addition, the hardware section 40 may alsoindicate to a lens to change the size of the iris in the lens. It willbe appreciated that the hardware section 40 may perform a considerablenumber of function other than motion detecting.

The video manipulate stage 52 may manipulate an image to clarify theimage as by correcting for color or extracting facial features. This isespecially important when faces are in the image and the faces are to bematched against a database identifying a particular face. A similar typeof manipulation is provided by the stage 54 with respect to audioinformation such as when a person is speaking. The event generator 56matches the image from the stage 52 against the images in the database.This is important when the images are faces. The event responder stage58 provides a response depending on the matching or lack of matching ofthe acquired image from the stage 52 and the image in the database.Although the matching has been discussed with reference to faces, thematching can be with respect to any physical object or any perceivedstate independent of a physical object.

The event responder 58 acts upon the output from the event generator 56in accordance with the processing which is provided to obtain thedesired results. The platform user interface 60 provides a mechanism fortaking the information that the camera intelligent imaging platform 18sees and the platform 60 calculates that information and presents thecalculated information to the user. It also allows for the user toadjust the setting of the camera. The platform configuration 65 allowsthe user of the system to write code for customizing the camera toprovide the desired result. The kernel operating system 62 provides thebasic operation of the camera. It is well known in the art.

Although the stages 52-62 (even numbers only) and 65 constitutesoftware, they may be disposed in the hardware section 28 c, since theycontrol the operation of the main processing hardware 66. The mainprocessing hardware 66 is sometimes referred to in this application as a“main processor”. The main processor 66 is connected by the bus 25 tocommunication stages or channels in the intelligent imaging platform 18.The intelligent imaging platform 18 includes a subset of communicationchannels 70 (ethernet), 72 (serial) and 74 (firewire) in thecommunications arrangement 20. The channel 70 receives information froman Ethernet source. The channel 72 receives serial information from anexternal source. The channel 74 receives high speed information fromsoftware known as Firewire and communicates this information tohardware. The channels 70, 72 and 74 are representative of the differenttypes of information that may be acquired by the currently activecommunication channels in the intelligent imaging platform 18. Therepresentative channels such as the channels 70, 72 and 74 also receiveinformation from the main processor 66 and supply information to themain processor.

The intelligent imaging platform 18 in turn communicates through thecommunications network 76 to the central station 14. As shown in FIG. 1,the central station 14 is reprogrammable and can initiate reprogrammingof itself and any other reprogrammable stage. The central station 14 isshown as including a station user interface 80, a station configuration82, storage 84 and a platform setup, programming and control 86. Theplatform setup 86 may include set up and configuration information forevent generation, event response, platform configuration, platform userinterface, field programmable gate array and coprocessor correspondingto what is shown in the intelligent imaging platform 18. The platformset up 86 is shown as being included in the central station 14 but itcontrols the state of the stages 34, 36, 52-62 (even numbers only) andthe main processor 66 in the intelligent imaging platform 18.

The stage 30 acquires the image from the camera and introduces theacquired image to the field programmable gate array 34. The gate array34 clarifies the image in accordance with the desired processing to beprovided of the image and introduces the signals representing theclarified image to the coprocessor 36. The coprocessor 36 manipulatesthe clarified image dependent upon the desired result to be obtainedfrom the system shown in FIG. 1. For example, the coprocessor 36 maymanipulate the image to focus on an individual in a crowd and may trackthe movements of the individual. Alternatively, the coprocessor maymanipulate the image to concentrate on what happens in a particularcorner of a room. The coprocessor 36 is also able to manipulate theaudio from the codec 32 to conform to the manipulation of the video.However, as indicated previously, the audio information may be clarifiedby the field reprogrammable gate array 34 before it is introduced to thecoprocessor 36.

The signals from the coprocessor 36 are further manipulated by thestages 52 and 54. The video manipulator 52 further enhances the qualityof the acquired image. For example, the video manipulator 52 mayidentify individual faces in a crowd and may extract facial features ofan individual. The event generator 56 may match the facial featuresagainst a database to identify the individual on the basis of thismatching against the database.

The system 10 shown in FIG. 1 has certain important advantages. Itprovides software (e.g. the stages 34, 36, 52-62 (even numbers only) and65 ) in the camera to control the operation of the camera. In this way,the reprogramming of the software stages can occur instantaneously inthe camera and the resultant changes in the output from the softwarestages can directly control the operation of the camera. This is incontrast to the prior art where output signals have been introduced tocontrols outside of the camera. In the prior art, the signals are thenprocessed outside the camera and introduced to the controls outside ofthe camera. As will be appreciated, the passage of the signals outsideof the camera and the subsequent processing of the signals outside ofthe camera produce a degradation in the camera performance.

The degradation of the signal resolution with increases in distance isparticularly troublesome when analog signals are processed many of thecamera systems of the prior art processed analog signals. In contrast,the system of this invention operates on a digital basis. Coupled withthe disposition of the camera controls in the camera, the digitaloperation of the system of this invention enhances the sensitivity andthe reliability of the operation of the system 10.

The system 10 also has other advantages. This results in part from theflexibility in the construction and operation of the system. Forexample, all of the stages 34, 36, 52-62 (even numbers only) and 65 arereprogrammable. Furthermore, each of the stages 34, 36, 52-62 and 65 canbe reprogrammed on the basis of a decision from that stage or any of theother of these stages. This flexibility in reprogramming provides for anenhanced sensitivity and reliability in the adjustments that can beprovided in the operation of the camera, thereby providing an enhancedperformance of the camera.

FIG. 2 illustrates the software in additional detail. It includes thevideo manipulator 52, which is shown in broken lines 90 in FIG. 2. Asshown in FIG. 2, the video manipulator 52 includes a preprocessor 92 andan analyzer 94. The preprocessor 92 converts the acquired image from thestage 30 in FIG. 1 to a format that the user wishes to provide. Forexample, the preprocessor 92 may correct, fix or establish colors in theacquired image or may select only a small portion of the image. Theanalyzer 94 may illustratively look for something specific in the imageor in a portion of the image. For example, the analyzer 94 may look foran individual having a particular identity. Alternatively, the analyzer94 may extract facial features or may detect motion of an image or anobject. The operation of the event generator 56 and the event responder58 have been indicated previously in connection with FIG. 1.

The output of the analyzer 92 is stored or archived as at 96 in FIG. 2and the stored or archived output is introduced to a post processor 98.The post processor 98 illustratively provides for a modification of theimage based upon the output of the analyzer 96 and the event responder58. For example, the post processor 98 may emphasize image portions thathave changed in position with time. The output of the post processorstage 98 is introduced to one of the stages 70, 72 and 74 in thecommunications network 20 in FIG. 1 and the output of the communicationsstage is provided to the communication network 76 also shown in FIG. 1.

The miscellaneous input/output stage 40 in FIG. 1 is also shown inadditional detail in FIG. 2 within a block 106 in broken lines. Themiscellaneous input/output stage 40 includes miscellaneous inputs 108such as triggers and includes miscellaneous outputs 110 such as relays,light emitting diodes and an iris control port for the camera. The audiomanipulator 54 in FIG. 1 is also shown in FIG. 2 within a box 100 formedfrom broken lines. The audio manipulator 100 in FIG. 2 includes apreprocessor 102 and an analyzer 104 which respectively operate on theaudio in a manner similar to the operation of the preprocessor 92 andthe analyzer 94 on the video in FIG. 2.

FIG. 2 also includes the platform user interface 60 and the kerneloperating system 62 shown in FIG. 1. The platform user interface 60includes commands and web pages and the kernel operating system 62includes timer tasks and load control. The “load” refers to the work inperforming the software tasks on the processor and the “load control”refers to the acts of organizing the tasks to make certain that all ofthe tasks are provided with an opportunity to occur. FIG. 2 alsoincludes the platform configuration 65 also shown in FIG. 1. Theplatform configuration 65 includes code load, set-up, original equipmentmanufacturers (OEM) compiles and the software development kit (SDK). Theplatform configuration 65 and all of the other stages in FIG. 2 includethe diagonal line 64 to indicate that each of the stages can talk to anyof the other stages in FIG. 2 and can be reprogrammed by the mainprocessor 66 on the basis of an initiation by any of the reprogrammablestages shown in FIG. 2.

FIG. 3 shows the acquisition of an image illustratively in eitherphotonic (light) or analog form. FIG. 3 also indicates the flow of videoand audio data through the hardware shown in FIG. 1. The video pathincludes the image acquisition of light illustratively in analog form asat 30 in FIGS. 1 and 3. The image is converted inside the image acquirer30 into digital form either by using a combination of a lens and imager(for light) or by using an analog decoder—for example, an NTSCdecoder—(for analog). The resultant signals flow through the video bus31 to the field reprogrammable gate array 34 also shown in FIG. 1. Thegate array 34 also receives audio signals flowing through a bus 122 fromthe audio codec or acquirer 32 also shown in FIG. 1.

The video and audio signals then flow through a bus 124 to thecoprocessor 36. The output from the coprocessor 36 is provided to a bus126. These signals then pass through the gate array 34 to PCI bus 38.The signals on the buses 120 and 122 may also be by-passed, through thefield reprogrammable gate array 34 to the PCI bus 38 without passingthrough the coprocessor 36. The signals on the PCI bus 38 pass throughthe main processor 66 and through a communications bus 129 to thecommunications network 76 in FIG. 1. As can be seen, the audio/videodata flows through as many as five (5) different buses but only oncethrough each bus. This allows for a streamlined flow of data through theintelligent imaging platform 18.

FIG. 6 is a simplified block diagram of the system shown in the previousFigures. In this Figure, the stages discussed previously in connectionwith FIG. 1 and considered as primary are shown. The simplified systemincludes the image acquirer 30 (designated as A), the field programmablegate array 34 (designated as B), the main processor 66 (designated asC), the software development kit 65 (designated as F), the centralstation 14 (designated as D) and the coprocessor 36 (designated as E).The software development kit F may be considered as a part of theplatform configuration 65 in FIGS. 1 and 2 and is included within themain processor 66.

FIG. 6 also includes a chart showing primary combinations of individualones of the stages A-F in FIG. 6 and optional combinations of theprimary stages with the other stages shown in FIG. 4. As will be seen,there are two (2) primary combinations—(1) a combination of A (the imageacquirer), C (the main processor) and F (the software development kit)and (2) a combination of A (the image acquirer), C (the main processor)and E (the coprocessor). Certain optional combinations are also showninvolving individual ones of B (the field reprogrammable gate array), D(the central station) and F (the software development kit) for theprimary combination of A, C and E. They further include optionalcombinations of B (the field reprogramable gate array), D (the centralstation) and E (the coprocessor) for the primary combination of A, C andF. The combinations designated with a star in the first column may beconsidered as the most critical. It will be appreciated that thecombinations shown in FIG. 6 are illustrative only and that aconsiderable number of other combinations (some even primary) may beprovided without departing from the scope of the invention.

FIG. 4 illustrates the configurability of different ones of applicants'primary stages. The stages correspond to the primary stages shown inFIG. 6—namely, the image acquirer 30, the field programmable gate array34, the coprocessor 36, the main processor 66, the software developmentkit 65 (within the main processor 66) and the central station 14. All ofthese stages (except for the image acquirer 30) are configurable orreprogrammable as indicated by a cloud like configuration within arectangular block. Each cloud represents a configurable orreprogrammable entity which can be shaped to the task at hand. Eachrectangular block represents the encompassing fixed body which is notunto itself configurable or reprogrammable.

The different blocks are defined and determined by the interfaces ofapplicants' assignee. These interfaces are as follows:

-   -   1. The video interface 31 between the image acquirer 30 and the        field reprogrammable gate array 34;    -   2. A coprocessor interface 142 between the gate array 34 and the        coprocessor 36;    -   3. The hardware interface 38 between the gate array 34 and the        main processor 66;    -   4. A communications interface 146 between the main processor 66        and the central station 14; and    -   5. A software interface 148 between the main processor 66 and        the software development kit 65.        The request to initiate reprogramming of a reprogrammable block        can come from anywhere in the system with the exception of such        stages as the image acquirer 30, the audio acquirer or codec 32        and the miscellaneous input/output stage 40. However, the        reprogramming is provided by the main processor 66.

FIG. 5 schematically illustrates in some detail a customizable block,generally indicated at 150, which is also representative of otherblocks. The customizable block 150 is reprogrammable as indicated by thediagonal arrow 64 in the lower right corner. The customizable block 150includes a sub-block 152 capable of performing a plurality of availablegeneric operations designated in the sub-block as tasks I to n. Theseoperations are generic to the block 150 and other blocks. When the block150 provides only available generic operations such as in the block 152,the block is not reprogrammable. The customizable block 150 may alsoinclude a sub-block 154 providing custom defined operations. These areoperations individual to the block 150. The sub-block 154 may provide Ito n custom operations. When the block 150 can provide one (1) or morecustom defined operations, the block 150 is said to be reprogrammableand is demarcated or indicated by the arrow 64.

FIG. 7 is a chart showing examples of different functions capable ofbeing performed by the system 10. It will be appreciated that FIG. 7shows only a few of the multitudinous operations that can be performedby the system 10. The first (designated as “Main Function”) column inFIG. 7 indicates four (4) different functions which can be performed bythe system 10. These four (4) functions are:

-   -   (a) “Remote Color Video Monitor with Archive”,    -   (b) “Face Print Generation and Upload to Server”,    -   (c) “Gun Shot Detection and Server Notification,” and    -   (d) “Subject Tracking with Realtime Video Monitor.”

The second column in FIG. 7 is designated as “Other/Combined Functions”.It indicates other functions which can be performed in addition to the“main function” specified in the first column. For example, anotherfunction or a sub-function such as “Audio Monitoring” can be performedin addition to the main function of “Remote Color Video Monitor withArchive.” As another example, other functions or sub-functions such as(a) “Generate Face Present Audio Alert”, (b) “Recognize Face and AlertServer of Match” and (c) “Recognize Multiple Faces Simultaneously” canbe performed with the main function “Face Print Generation and Upload toServer”.

The third column in FIG. 7 indicates the function that is performed inthe preprocessor 92 in FIG. 2. For example, the preprocessor 92 performsa color correction when the main function is “Remote Color Video Monitorwith Archive”. As another example, the pre-processor 92 provides a“Facial Feature Extraction” when the main function is “Face PrintGeneration and Upload to Server”. The operation of the third (3^(rd))column in FIG. 7 is dependent on the operation of the first and secondcolumns of FIG. 7. This is also true of the operation in the fourth(4^(th)) through eleventh (11^(th)) columns of FIG. 7.

FIG. 7 indicates the operation of the analyzer 94 in the fourth columnof FIG. 7 for different ones of the main functions in column 1 of FIG.7. For example, for the main function of “Face Print Generation andUpload to Server”, the analyzer 94 operates to provide Face PrintGeneration. As another example, for the main function of “SubjectTracking with Realtime Video Monitor,” the analyzer 94 operates toprovide “Subject Detection/Motion Estimation.”

Column 5 in FIG. 7 indicates the operation of the event generator 56 inFIGS. 1 and 2 for the different main functions in column 1 of FIG. 7. Inlike manner, column 6 in FIG. 7 indicates the operation of the eventresponder 58 in FIGS. 1 and 2 for the different main functions in column1 of FIG. 7. Similarly, column 7 in FIG. 7 indicates the operation ofthe storage member 96 in FIG. 2 for different main functions in column 1of FIG. 7. The post processor 98 in FIG. 2 provides the operations shownin column 8 of FIG. 7 for the different main functions specified incolumn 1 of FIG. 7. The communication stages 70, 72 and 74 in FIG. 1perform the operations shown in column 9 of FIG. 7 when the mainfunction is as indicated in column 1 of FIG. 7 and the other combinedfunctions are as indicated in column 2 of FIG. 7.

Column 10 of FIG. 7 indicates the miscellaneous output which is providedwhen the main function specified in column 1 of FIG. 7 is provided. Themiscellaneous output is indicated at 40 in FIG. 1. The audiopreprocessor 102 in FIG. 2 is shown in FIG. 2. The audio preprocessor102 in FIG. 2 provides the operations shown in column 11 of FIG. 7 forthe different main functions specified in column 1 of FIG. 7. In likemanner, the audio analyzer 104 in FIG. 2 provides the operations shownin column 12 of FIG. 7 for the different main functions specified incolumn 1 of FIG. 7. The last column of FIG. 7 is designated as “Notes”.This column indicates that the field programmable gate array 34 and thecoprocessor 36 are utilized for added processing power when “RecognizeMultiple Faces Simultaneously” is provided as a sub-function in column 2of FIG. 7. As another example, the gate array 34 and the coprocessor 36are utilized when “Multiple Subject Tracking is Provided with DigitalPan, Tilt, Zoom” is provided in column 2 of FIG. 7.

Although this invention has been disclosed and illustrated withreference to particular embodiments, the principle involved aresusceptible for use in numerous other embodiments which will be apparentto persons of ordinary skill in the art. The invention is, therefore, tobe limited only as indicated by the scope of the claims.

1-73. (canceled)
 74. In a combination for processing an image to providea desired result, a preprocessor for concentrating upon particularcharacteristics of an event, an analyzer for analyzing the particularcharacteristics of the event to identify individual characteristics ofthe event, an event generator for making decisions involving theindividual characteristics upon the identification of the individualcharacteristics of the event, an event responder for providing actionwith respect to the individual characteristics of the event inaccordance with the decisions made by the event generator, and mainprocessor for providing a controlled operation of a camera in processingthe image in accordance with the responses provided by the eventresponder.
 75. In a combination as set forth in claim 74, thepreprocessor, the analyzer, the event generator and the event responderbeing reprogrammable to adjust the individual characteristics of theimage provided by the camera.
 76. In a combination as set forth in claim74 wherein the preprocessor, the analyzer, the event generator and theevent responder are communicative with any other one of thepreprocessor, the analyzer, the event generator, the event responder andthe main processor to provide for an adjustment in the operation of anyone of the preprocessor, the analyzer, the event generator and the eventresponder with which the main processor is communicating.
 77. In acombination as set forth in claim 74, a post-processor for modifying thesignals representing the image based upon the operation of the analyzer,the event generator and the event responder and for introducing themodified signals to the main processor to facilitate the processing ofthe acquired image by the main processor.
 78. In a combination as setforth in claim 77, a camera responsive to the operation of teh mainprocessor for providing the desired signals.
 79. In a combination as setforth in claim 79, each of the preprocessor, the analyzer, the eventgenerator, the event responder and post-processor being reprogrammableand the reprogramming of each of the preprocessor, the analyzer, theevent generator, the event responder and the post-processor beinginitiated by any one of the preprocessor, the analyzer, the eventgenerator, the event responder and the post-processor and the mainprocessor.
 80. In a combination for processing an image, an imageacquirer fro acquiring the image and for converting the acquired imageto signals representing the image, an event generator responsive to thesignals from the image acquirer for prioritizing individual ones of thesignals in accordance with the results desired to be obtained in theprocessed image, an event responder responsive to the prioritizedsignals for making decisions based upon the prioritized signals from theevent generator and upon the desired results and for providing signalsrepresenting these decisions, and a hardware processor for processingthe signals representing the decisions from the event responder toobtain the desired results.
 81. In a combination set forth in claim 80,each of the event generator and the event responder being reprogrammableby the hardware processor in accordance with the results desired to beobtained in the acquired image.
 82. In a combination as set forth inclaim 81, a video manipulator responsive to the signals from the imageacquirer for modifying the signals representing the acquired image toclarify the image represented by the signals and for introducing themodified signals to the event generator, the video manipulator beingreprogrammable by the hardware processor in accordance with the resultsdesired to be obtained in the acquired image.
 83. In a combination asset forth in claim 80, an audio acquirer for acquiring audio informationrelated to the image and for converting the audio information to signalsrepresenting the audio information, and an audio manipulator responsiveto the audio signals for modifying the audio signals to clarify theaudio information represented by the signals, The audio manipulatorbeing reprogrammable by the hardware processor in: accordance with theresults desired to be obtained in the acquired image.
 84. In acombination as set forth in claim 81, a camera, the image acquirer, theevent generator, the event responder, the hardware processor and thevideo manipulator being disposed in the camera.
 85. In a combination asset forth in claim 83, camera the image acquirer, the event generator,the event responder, the hardware processor, the audio acquirer and theaudio manipulator being disposed in the camera. 86-111. (canceled) 112.In a combination as set forth in claim 83, a camera the image acquirer,the event generator, the event responder, the hardware processor theaudio acquirer and the audio manipulator being disposed in the camera.113. In a combination for processing an image to provide a desiredresult, a first stage for identifying individual characteristics of anevent, an intelligent imaging platform for indicating the individualcharacteristics of the event, an event responder for changing theindividual characteristics of the event, a post processor for providingchanges in the image in the intelligent imaging platform in accordancewith the changes indicated in the event responder.
 114. In a combinationas set forth in claim 112 wherein the first stage, the event responder,the intelligent imaging platform and the post processor arereprogrammable.
 115. In a combination as set forth in claim 113 whereineach of the first stage, the event responder, the intelligent imagingplatform and the post processor is reprogrammable by any of the othersor by itself.
 116. In the combination set forth in claim 114, a databasefor providing a desired image, an event generator for providing a matchbetween the individual characteristics of the event and the desiredimage in the database, and the event responder providing a responsedepending upon the matching or lack of matching of the individualcharacteristics of the event from the first stage and the desired imagein the database.
 117. In combination for processing an image to producea desired result, a video manipulator for enhancing particularcharacteristics of the image, an event generator for concentrating on atleast an individual one of the particular characteristics of the image.an intelligent imaging platform for indicating the image with theconcentration on at least the individual one of the particularcharacteristics in the image, and a main processor including the eventgenerator, the intelligent imaging platform for providing an imageconcentrated on the at least individual one of the particularcharacteristics in the image.
 118. In the combination recited in claim116 wherein video manipulator and the event generator are in theintelligent imaging platform.
 119. In the combination recited in claim116, including a database for indicating the desired image, and theevent generator being operative to match the image in the videomanipulator with the desired image in the database, and an eventresponder for providing a response depending upon the matching or lackof matching of the image in the video manipulator with the desired imagein the database.
 120. In combination for processing an event to providea desired result, a first stage for identifying individualcharacteristics of the event, an event responder for processing theindividual characteristics of the event, and an intelligent imagingplatform for indicating the event as processed by the event responder.121. In a combination as set forth in claim 119 wherein the first stage,the event responder and the intelligent imaging platform arereprogrammable.
 122. In a combination as set forth in claim 120 whereinthe first stage, the event responder and the intelligent imagingplatform are self reprogrammable.
 123. In a combination as set forth inclaim 119 wherein the image provided by the first stage is matchedagainst images in a database and the event responder provides a responsedepending upon the matching or lack of matching of these images andwherein a platform user interface processes the response from the eventresponder for introduction to a user.
 124. In a combination as set forthin claim 120 wherein the image provided by the first stage is matchedagainst images in a database and the event responder provides a responsedepending upon the matching on lack of matching of these images andwherein a platform user interface processes the response from the eventresponder for introduction to a user.
 125. In a combination as set forthin claim 123 wherein each of the first stage, the event responder, theintelligent imaging platform and the platform user interface isreprogrammable by anyone of the first stage, the event responder, theintelligent imaging platform and the platform user interface.
 126. Incombination for processing an image to provide a desired result to auser, including, a preprocessor for manipulating the image to clarifythe image, an event generator for matching the manipulated image againstimages in a database, an event responder for processing the output fromthe event generator, and platform user interface for processing theinformation from the event generator and presenting the processedinformation to the user.
 127. In a combination as set forth in claim 125wherein each of the preprocessor, the event generator, the eventresponder and the platform user interface is self reprogrammable and isprogrammable by the other ones of the preprocessor, the event generator,the event responder and the platform user interface.
 128. In acombination as set forth in claim 125, including, an intelligent imagingplatform for displaying the image, and a software development kitresponsive to the output of the event generator for customizing theoutput of the intelligent imaging platform.